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| System and Method for Indexing Recordings of Observed and Assessed Phenomena 
| Using Pre-defined Measurement Items 

This application is a continuation in part of provisional patent application serial no. 
60/127,194 on March 31 , 1999, the disclosure of which is herein incorporated by reference. 
| This invention was mad esupported with a grant from the U.S. government which 

may have rights to the invention. 

Copyright Notification 

Portions of this patent application contain materials that are subject to copyright 
protection. The copyright owner has no objection to the facsimile reproduction by anyone 
of the patent document or patent disclosure, as it appears in the Patent and Trademark Office 
patent file or records, but otherwise reserves all copyright rights whatsoever. 

Field of the Invention 

The present invention g e n er ally relates to information processing systems . More and 
observation and assessment methods and operations, including the training and monitoring 
of interviewers or observers, quantitative and qualitative rating and assessment processes, 
and the storage and retrieval of recordings of interviews or observed events and, more 
particularly, the invention is a data captur e and access system that correlat e s data with 
recorded signals with an index based on prc-identifi e d identifiers at the time of capture but 
allows later annotation or additional ind e xing without affecting the original index identifiers 
| or the recorded signals. The invention specifically provides at o_ajsystem and method for 
| indexing of digital recordings of events bv us e of pre-entcrcd or pre-existin g recordings of 
| interviews or other observed events or phenomena by combining systematic use by an 
| observer of pre-defined descriptive or quantitative measurement items that serve as index 

information designators such as key words o r labels. Such a method is particularly useful as 
| those used in a systematic intervi e w or assessment method, w ith simultaneous digital 
| recording to automatically generate indexes. 
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Many events ii n the medical, legal, scientific, business, and education domains, 
among others, today arc being digitally record e d. Physicians may r cco r d it is necessary to 
systematically observe and describe, assess, rate, or otherwise measure events or processes 
(i.e.. phenomena). It is also useful to make recordings of these events and to be able to 
5 playback these recordings. For example, physicians may use audio, video, or 

electrophysiological recording of an interview, a physical examination, a diagnostic 
procedure, an operation, or autopsy a and may need to annotatc retrieve and examine specific 
parts of that recording at some later time to notes, obse r vations, or other content in o r de r to 
make a more complete record or draw previously unseen trends. Lawyers may s ee k to for 

10 various purposes. Attorneys may carry out and record a witness deposition and later 

Hn kplavback sections of that oral testimony in order to link the testimony to other evidence 
(documentary, video, or testimony by other witnesses) in order to draw parallels-or s 
highlight inconsistencies , or for other purposes. Social scientists apply systematic 
observation and rating methods to study society and culture . The frequency of such 

15 recordings, audio, video, or other, is likely to increase as their usefulness continues to grow 

the r eb y . In general, the usefulness depends upon the purposes and value of retrieval and 
playback, the range of retrieval and playback functions, and how accurately and efficiently 
retrieval and playback can be carried out. Purposes include: providing an educational or 
teaching tool to illustrate observation, assessment, and measurement techniques in general. 

20 creating teaching materials; monitoring teachers, trainees, and raters; allowing for review of 

the assessor's conclusion or diagnoses , p r oviding an educational or teaching tool to illust r ate 
optimum assessment interview techniques, documenting the ; supporting retrieval of multiple 
interviews, events, or segments of interviews or events for comparison and other study; 
reviewing an interview for legal and/or other purposes ; and other similar and varied uses in 

25 many domains r 

MAs a more detailed and prototypical example, m edicine and public health are 
domains in which considerable effort has been devoted to the development of tools fe.g., 
questionnaires or assessment schedules and instruments} for conductin g systematic 
assessment interviews and examinations . Th e A main objective of these efforts has often 

30 been to improve the accuracy, reliability, and efficiency of a clinician's diagnosis of his/hc ra 

patient. 
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Among medical disciplines, psychiatry researchers have perhaps allocated the most 
efforts to developing patient assessment akt smethods and tools . Examples of the types of 
assessment instruments that have been develope d, which are composed of descriptor or 
quantitative assessment (i.e., measurement) items or questions, include the Structured 
Clinical Interview for DSM (SCID), the Diagnostic Interview Schedule (DIS), the 
Composite International Diagnostic Interview (CIDI), the World Health Organization 
(WHO) International Personality Disorders Examination (IPDE), and the WHO Schedules 
for Clinical Assessment in Neuropsychiatry (SCAN). 

These tests have been standardized around a core setg of inquiries or 
directions m easurement items. Measurement items typically are composed of: 1) definition 
of a phenomenon of interest; 2) instructions regarding observation of the phenomenon; and 
3) a quantitative rating scale. The scientific field of measurement known as psvchometrics. 
includes study of and methods pertaining to systematic and standardized observation, 
measurement, training, monitoring, and comparison. Psychometric issues of importance 
include efficiency, reliability (consistency), and validity (accuracy) . 

The response to or rating of standardized questions , inquiries, or 
di r ections measurement items and instructions is of interes t for th e ass e ssor , both in terms of 
the substantiv e re s p ons e assessment results as well as the behavior manifested in the delivery 
of the assessment and ofin the response. PAs well as expressed symptoms, p sychiatrists 
may want to measure or rate the behavioral responses of patients , o r may wish to observe 
and rate t he actions o f other psychiatrists in training. Cardiologists may want to measure 
physical responses to changes in applied stress, such as th echanges in EKG readings during 
a cardiac stress test. Lawyers may seek to identify inconsistencies between witnesses to an 
accident using standardized questions. Similar examples are found in virtually every 
profession where a pattern of systematic observation and testing for response^ and analysis 
of th e response is used. 

However. t hat response, is used. In all these fields, analog or digital recordings of 

audio, video, physiologic, or other aspects of observed events are used to assist in these 
processes. 

However, most anyone who has participated in r eviewing these retrieving and 
organizing interview or testing reeord srecorded materials for review and study realizes that 
the review process needs help and imp r ov e ment. If there was an e ffici e nt way to index the 
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video, audio or other type of r e cording when the interview was r eco r ded, much time could 
be saved when later searching the reco r ds for particular info r mation. 

It is, therefo r e, an object of the present invention to provide a system and method for 

conducting, r eco r ding, nlaving back and editing o r annotating digital recordings in r etrieval 
5 and review process is usually tedious, often incomplete and inaccurate, and has vast room 

for improvement. What is needed for improved retrieval and management of audio, video, 
or other format. 

Kevt orecorded materials are indexes that are meaningful accurate, and easily 

generated, manipulated, and extended. Indeed, one of the most important features of any 

10 useful recording and retrieval system based on digital media is the ability to efficiently 

generate meaningful indexes for the recorded material. Such indexes ar e well suitcd would 
be especially valuable for structured or systematic information gathering events that use a 
syst e matic question-and-answer method containing a s e t or list of standardized inqui r ies, 
e .g., a list o f processes in which professional or scientific standards are required and/or in 

15 which considerable efforts are made in training, quality control, and other administrative 

functions, e.g., a television interview based upon a list of questions, giving a patient a health 
questionnaire, a clinical or research medical assessment, conducting a-set-of deposition 
questions, , using a list of actions to activate brain function in different ways, an dconducting 
focus groups, ethnographic research, or administering a graded cardiac stress test, among 

20 others. 

In the fields of medicine and health, the established means of recording structu r ed 
assessments has generally been by use o f analog audio or video tape recorders. Indexing, if 
used at all, has been generally t hrough means such as length counters and manually added 
entries , or with hybrid combinations of tape recorders with microprocessor systems . This 

25 mademakes the process of creating indexes t edious and often incomplete. In addition^eyen 

after indexes are created, the use of tape recorders means that the process of searching and 
playback tsremains slow and inefficient. Finally, creation of generally meaningful indexes 
has been problematic. 

In the field of digital recording, many indexing schemes have been developed, 

30 ranging from low-level encoding to linking of text with video frames. Industry standard 

digital encoding methods such as the MPEG-2 (Society of Motion Picture Engineers) 
format, include channels for indexing based upon time code. However, most of these 
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approaches generate index entries in a form not directly useful for retrieval p layback, and 
management, but rather provide basic mechanisms for supporting indexing. 

-Qn cFor example, one prior art system is known that uses computer voice-to-text 

processing to automatically generate words that are then used to index the source digital 
audio recording. Although ttscftrfof some use , it does not provide a more organized level of 
indexing, since many words will be used in many areas of a recording, and hence as such do 
not provide specific indexing to unique areas or segments of a recording. For indexing to be 
more useful, the indexes or index items should contain meaningful and unique information 
that is associated with th eunique recording locations or segments. AManv indexing 
methods can support this, but it is generally tedious and inefficient to generate such index 
items manually. 

Existin g Examples of prior art indexing systems support some forms of indexing of 
recorded signals in a variety of ways, include U.S. Pat. No 4.841 .387. which discloses an 
indexing system using a touchpad to correlate simultaneously recorded audio or video 
information to handwritten notes made on numbered sheets of paper manually placed on top 
of the touchpad. This system is simply a means for replay of recorded audio or video 
information through manual entry of sheet numbers and physical contact with the paper in 
the approximate area of the original handwritten note, to generate a touchpad signal that is 
used to access the correlation indexed recording. The system does not anticipate correlation . 
of recordings with any information other than the position of handwritten notes and a page 
number, nor does it provide any other means of identifying, searching, or retrieving 
recorded materials, nor does it support psychometric and more general observation, 
measurement, retrieval, training, monitoring, or comparison functions and operations. In the 
professional and technical arenas, it is systematic as well as quick access to specific 
passages or segments of recorded material that determines a system's usefulness, which is 
not adequately provided bv utilizing the position of handwritten notes for indexing. 

European patent application publication EP 0 495 612 by Lamming discloses a 
com p ute r -based not e -takin g n indexing system int e grated with a nt hat connects manually 
entered computer notes to audio or video r eco r ding system. Hie computer presents a 
document editor styl e user interface to the user who eithe r creates a new document or 
ret r ieves an existing document to which the user adds notes as a recording is made or played 
via the integrated audio or video system. As th e use r enters each note (mark or indicum), 
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the indicum is added to the document and it is time stamped and stored in an information-to- 
time-stamp index* The time stamps are not visible to the user; they are stored with the 
computer's inte r nal r e presentation of the indica entered by the user. A video-frame time 
stamp function time stamps time code data received from the audio or vid e o recorder and 
5 . . c r eates a time stamn-to-timc^cod e index. A browser function recordings, and p ermits the 

user to retrieve sections and play sections of a recording bv selecting a note. The indexing is 
based upon spatial location and organization of a set of notes within a document rather than 
the meaning of the notes, requiring the user to manually associate notes bv physical location 
within the electronic document. Because of this limitation, when a user wishes to associate 

10 a segment of the recording using the indicia directly by selecting the indicia. The browse r 

looks up the indicia in the first index to retrieve the tunc stamp, and looks up the time code 
of the r ecording in the second index using the tim e stamp, playing the s e ction of the 
recording in the area indicated by the tim e code. EP O 495 612 also discloses how time 
stamping the indicia may be applied to cr e ating topic or key wo r d data. By entering new, 

15 separate indicia s p atially near a with an earlier note, the user must navigate to the page and 

location in the existing set of notes, presenting a serious inefficiency with lengthy notes. 
This system is also limited with regard to real-time augmentation of previously entered 
indicia that is a key word or topic whenever an idea or speak e r or topic a p plies to the 
pr e viously entered indicia, later selection of all of the marks spatially associated with a topic 

20 will result in all sections of the recording indexed by the time stamps of the respective 

indicia to be replayed. 

notes during recording, since the later notes will be time stamped with the time they 

were entered rather than with the time of the earlier material to which they relate or are 
relevant, lacking any automatic index linkage to the earlier material. As with U .S. Pat. No 

25 4, 84 1,387, e ntitled "Arrangement fo r Reco r ding and Indexing Information" and issued to 

Rindfuss, discloses a system for reco r ding information r e lating to an event on a recording 
medium, such as an audio or video tape, and for indexing positions of handw r itten notations 
made on a touch sensi t ive device and concerning the event to positions on the r ecording 
medium in o r de r to allow the use r to identify portions of the handw r itten notations for which 

30 review of the co r related material on the r e corded medium is desi r ed. In the recording mode, 

the device makes an audio recording of the event on a standa r d cass e tte tape. 
Simultaneously, the elect r onic touchpad senses the position of the user's handwritten notes 
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on th e w r iting surface, and provides this information to the mic r op r ocessor which cor r elates 
th e record of the positions of the handw r itten notations on each page with the position of the 
r ecorded information on the audio tap e at cor r esponding instants in time. Real-time 
const r ained correlation vectors r e pr e senting rec t angular areas of the dis p lay surface each 
containing a cohesive group of handw r itten text are each combined with a tape position that 
correlates to the instant in time the handwriting within that area began. this system requires 
manual entrvof notes, does not standardize the process of note-taking nor provide any 
systematic means of observation, description, or assessment, does not combine systematic 
assessment with generation of indexes, does not provide a means for quantitative rating of 
observed events, and does not provide a means of retrieval and playback of recorded 
materials combined with systematic re-assessment or monitoring, 

U.S. Pat. No. 4,425 5, 586 issued to Mille r l72,281 discloses a system that combines a 
vid e o tape recorder with a computer in such a manner that these two components e ach 
automatically reco r d and display whe r e related information is stored in its own m e chanism 
as well as in its counterpart mechanism, allowing the use r to det e rmine the location of all 
the corresponding data stored both on video tape and on a storage medium such as a 
disk e tte, by examining only one storage medium. Notes about the r ecorded event o r 
document may be entered onto th e diskette along with the automatic e ntry of the 
corresponding reel number and frame number of the video record and diskette and file 
address number of the computer storag e medium. Another feature disclosed is the 
capability of the syst e m to enter and display the time and date on both the video tape and 
diskette r ecording mediums as well as on both vid e o monitors along with the data address 
location information. supports an operator in manually correlating previously recorded video 
material with a standard text transcript for the purpose of generating an index that can then 
be used bv software that searches text and finds associated video indexes. This system has 
no capacity to operate in real-time situations. It does not provide means for automated 
definition of segments of recorded material, nor does it provide an automated means of 
generating the actual indexes. It searches the text material only based upon operator-entered 
key word comments, and is unable to take advantage of all the information in the original 
text. It is subject to the problem of specific comment words being used for more than one 
indexed timecode, requiring the operator who generates the indexes to keep track of and 
exercise judgement about the choice of comment words, a significant weakness because it 
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does not provide a means of handling multiple occurrences of index key words in the | 
transcript material. I 

U.S. Pat. No. 4,924,387 issued to Jcppcson discloses a computerized court reporting 
system wh i ch provides for pe r iodically annotatin g discloses a system by which a | 
5 stenographer can control a video camera so as to stenographically record a deposition while | 

simultaneously operating a video camera and recorder. This system's main objective is to | 
provide a system to accurately record a deposition, not to efficiently and precisely locate | 
and present specific passages of the videotape deposition. It provides a limited indexing | 
structure since the stroke record made by the user of a court st e nog r aphic machine with a | 

10 time stamp from a system clock while simultaneously sending a time stamp to a r e cording 

system making a video and audio r eco r ding of the testimony. Th e logic of a control system 
determines automatically when to time stamp the stroke record and permits the user to 
trigger a control function to annotate the video recording with automatic "on the record" and 
"off the record" messages with associated time stamps. 

15 U.S. Tat. No. 5,564,005 issued to Webe r ct al. discloses a gene r al syst e m providing 

means for the user to first c r eate a ^timc zone" cor r elated to a recording of an e vent, and 
then to flexibly enter into the time zone information relevant to the event, addressing some 
of th e disadvantages of thc is intended to be a verbatim transcription of the verbal testimony | 
made in a courtroom, rather than any systematic observations or assessment of the | 

20 proceedings in the courtroom. I 

For all these systems described above: I 

These real-tim e data corr e lation and access systems have similar disadvantages. In 

th e case of U.S. Tat. No. 5,564,005, the system does not automatically generate meaningful 
indica at the time the "tim e zones" arc established by the actions of the user. Rathe r , 

25 information about the events correlated to a time zon e is entered later, in the form of 

handwritt e n strokes, or may be generated and entered in real-time, subject to the conceptual 
and processing speed limitations and varianc e of a user's abilities. Therefor, the primary 
initial index guide to the meaning of the events within a time zone arc the temporal 
f Wafffrrigt i f.g nf * hr timf ynur aiipmanted at the time bv the memory of the use r, the user's | 

30 ability to index notes or comments to the recording is entirely controlled through the indicia, | 

or notes, the user has manually entered in a document. For both real-time and post | 
processing, these systems require the user first to process mentally the recorded events and I 
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then generate and enter text to create notes, subjecting the system and its indexes to 
individual user variance in generating the text content. No method is suggested for 
facilitating or enhancing a user's personal note-taking style or accommodating the note- 
taking functions towards more standardized and systematic observation and assessment 
applications . In all these systems, no method or us e r int e rfac e d e sign is explicitly suggested 
for handling the case of repetition or extension of certain events or activities in th ea single 
recordin g or across multiple recordings, using a common indica for events or activities 
th e reby relat e d but occurring at different times, thereby facilitating or enhancing a use r 's 
personal note-taking style or accommodating the note-taking function to a variety of 
applications. For all except U.S. Tat. No. 5,564,005, the us e r's ability to ind e x notes to the 
recording is e ntir e ly controlled through the indicia, or notes, the user has e ntered in a 
document, since the time stamps or position captures arc those made at the time the notes 
are entered. Those systems assum e , th e refore, that the time of entry of a note sufficiently 
corresponds with the time o r positions of the reco r ding to p r ovide an adequate index into the 
recording, requiring the use r to mentally process the on-going r eal-time e v e nts and within a 
certain tim e varianc e g e n er ate and e nt er th e not e , hi the case of U.S. Pat. No. 4, 9 24,387, 
this provides an adequate indexing st r ucture since the stroke record made is intended to be a 
verbatim transc r iption of the verbal testimony made in a courtroom. In the case of EP 0 495 
612, however, where a verbatim transcription of the e vent may not b e the note-taker's 
intention, such an assumption do e s not allow fo r the later, real-time augmentation of 
previously ent e red notes, whil e still r e cording, with additional notes related to the 
pr e viously e ntered not e s, since the later not e s will be tim e stamped with the time th e y were 
entered r athe r than with the time of the mate r ial to which they re late o r are relevant. In the 
case of the syst e m disclosed by Rindfuss, notes may be entered later, but the later-ente r ed 
not e s will be correlated with the position of th e tape at the time the later not e s were entered, 
rath e r than with the position on the recording of the material to which they relate or arc 
relevant, hi the case of the syst e m disclosed by Miller, notes addressed to the diskette 
reco r d of the event at a later tim e are entered entirely during a post-processing phase and not 
during the r eal-tim e r e cording of the event. In some of these systems, neither time stamps 
no r tape positions a r e visible to the use r and so the temporal or spatial context of the ente r ed 
indicia is not availabl e to the user to enhance the r et r ieval function. In addition, in EP 0 495 
612 the function provided for creating topic or key wo r ds from the entered indicia may be 
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practically limited to one display u pagc" or scr e en unless the use r reenters the topic or key 
word on a second scr ee n or sc r olls b e tw ee n screens to add a mark to a p r eviously ente r ed 
topic or key word. As with augmentation of notes in general, the r e is no facility for 
associating a key word or topic name c r eat e d at a later time with notes ente r ed earlier, 
5 Existing post-p r ocessing (non-real-time) annotation systems in the field of post- 
production video editing provide fo r the creation of annotations about scenes cor r elat e d with 
"in" and "out" tim e codes identifying the scenes on a video recording. U.S. Pat. No. 
5,21 8,672 is an example of such a system. It is disclosed there that scene descriptions may 
be revised after initial cr e ation, but the correlation of the annotations are confined 

10 temporally to the identified scenes. 

In the post-processing system for the correlation of legal depositions with video 

r e cordings thereof disclosed in U.S. Tat. No. 5,172,281, a time code number is assigned by 
an operator of the system to both the computer transcript and the videotape segment where 
e ach question and answ e r passage begins, requiring manual time code entry. The location 

15 of individual words in the transcript may also be correlated with their corresponding 

position in the video recording, but there is no such result during the real-time process. In 
addition, the system do e s not appear to provide fo r the entry of not e s or annotations. 

Besntte related events or phenomena. 

Thus, despite the number of such prior art methods, therea need exists a-need-for a 

20 r e corded aa information system that will digitally record events , processes, or other 

observable phenomena, and simultaneously, efficiently, and automaticall v accurately 
generate informative indica that are correlated in r eal-tim e with the recording of the events 
without aff e cting an interviewe r ' s indexes in real-time, with minimal interference to the 
interviewer's or observer's ability to take notes or assess the flow of events. iThere is a 

25 more general need for systematic, structured, or standardized observation and assessment 

measurement methods, and there is a need to generate standardized, systematic, and 
quantitative indexes that are less subject to individual variance and which can be used 
between and among recorded information pertaining to different individuals or events so 
that comparisons and other analyses can be made. Furthermore, it would be desirable to 

30 have a system for annotatin gt hat allows subsequent annotation of an indexed digital 

recording of information in such a manner that the annotation did not disrupt the 
standardization of the index or otherwise adversely affect the ability of one set of recorded 
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information in a standardized index key to be compared to another set of recorded 
| information havm ethat uses the same standardized index key. 

There is also a need to gene r ate a standard set of index e s that arc not subject to 

individual variance and which can be used between and among reco r ded information 
pertaining to differen t individuals o r events so that comparisons can b e made, hi the case of 
video and audio logging tools, exclusively postprocessing systems arc inadequate for 
generating notes and r elationships between recorded segments, and are time consuming 
because they requi r e review of the entire tape in order to generate an index. Automatic 
indexing of video and audio notes by system time stam p ing of key- or handwritt e n strokes 
or automatic detection of s p eaker voice changes does not provide adequate context marke r s 

Objects and Advantages 

Accordingly, it is an object and advantage of the present invention to combine 
systematic, standardized observation and assessment using pre-defined descriptors or 
measurement items with simultaneous recording to automatically generate meaningful 
indexes for the recorded signals or does not permit user control of the amount of detail to be 
captured. For example, an index c r catcd r ecording in real-time. Additional objects and 
advantages are: ' 

- to provide a system and method that supports the generation and storage of new 
descriptor or measurement items with rating scales for subsequent use in systematic 
observation and assessment; 

- to provide a system and method that can import and store a set of pre-defined 
descriptor or measurement items, such as an existing questionnaire or set of rating scales, 
for subsequent use in systematic observation and assessment; 

- to provide a system and method that uses pre-existing descriptor or measurement 
items to structure the observation and assessment of events; 

- to provide a system and method that uses measurement items that employ 
quantitative rating scales; 

- to provide a system and method that efficiently creates an index for any type of 
recording bv fully automatic correlation with the use of pre-existing descriptor or 
measurement items; 
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- to provide a system and method that stores recorded material in digital form in 
centralized electronic storage; 

- to provide a system and method that stores descriptions or ratings of observed and 
assessed events or phenomena in centralized electronic storage: 

- to provide a system and method that allows unlimited repeated use of any 
particular pre-existing measurement item-in other words, an observer can repeatedly assess 
or rate a phenomenon or a recurring event using the same measurement item; 

- to provide a system and method that allows unlimited concatenation of indexed 
recorded material when one or more phenomena or recurring events are repeatedly assessed 
usinp: the same measurement item: 

- to provide a system and method that can use any part or parts of the content of pre- 
existing descriptor or measurement items as index labels or key words; 

- to provide a system and method in which a standard index, based upon a specific 
questionnaire or measurement instrument (le., a set of measurement items! can be applied 
across multiple recordings; 

- to provide a system and method that can search, sort, and/or retrieve recorded 
materials based upon meaningful words or combinations or words from the full text of the 
descriptor or measurement items used in the index; 

- to provide a system and method that can search, sort, and/or retrieve recorded 
materials based upon numerical ratings from standardized observation and quantitative 
assessment using measurement items with rating scales; 

- to provide a system and method that permits assessors to re-order recordings of 
events and compare similar events from different subjects on the basis of sp e ak er 
s e gm e ntation of the material would tell who was sp e aking but not the substance of the talk. 
Establishing time zones through the action of a use r do c s not di re ctly g e n e rat e indica 
containing info r mation regarding the events within the time zones. 

-the associated measurement items in order to investigate possible relationships and/or 
trends not previously discernible; 

- to provide a system and method that allows a user, such as a student or trainee, to 
view and/or review prior assessment results, i.e., descriptor or measurement item assessment 
choices, by the same user or bv a different user, in parallel with playback of the recording of 
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the observed and assessed events or phenomena, allowing for efficient use of actual 
assessment for teaching or training; 

^to provide a system and method that allows playback of recorded materials with 
the prior selections of descriptors or ratings hidden from display: 

^_to provide a system and method that allows re-assessment or re-rating during 
playback so that rater efficiency, accuracy, and reliability can be measured; 

- to provide a system and method that allows multiple users to make ratings based 
upon a single original recording, and that allows multiple users to playback and rate at the 
same time; 

- to provide a system and method that supports comparison of assessment results or 
ratings across multiple raters, subjects, events, or geographical sites, for analysis, teaching, 
monitoring assessment or other research purposes; 

- to provide a system and method that can automatically compare ratings to make it 
easier to teach, monitor, and conduct research; 

- to provide a system and method that allows unlimited manual addition of index 
points and labels during playback; and 

- to provide a system and method that allows assessors and others using a "test and 
response" method of investigation greater note-taking and annotation efficiency and 
flexibility than was previously available. 

Further objects and advantages of the invention will become apparent from a 
consideration of the drawings and ensuing discussion. 

Summary of the Invention 

It is an object of the invention to provide a system and method for digitally recording 
information about an event or individual and generating a standard set of indexes fo r this 
information. 

It is a further object of the invention to provide a system and method for annotating 

an indexed, digital recording of information without changing the standardized index 
markings in such a manner that would adversely affect the ability of one set of recorded 
information in a standardized index key to be compared to anothe r set of r eco r ded 
information having the same standardized index key. 
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In accordance with a preferred embodiment of t The present inventio n, the invention 

e mploys a system that includes: (a) means for pr e senting a subject o r ope r ator with a 
m e asur e ment item; (b) a r e corde r that re co r ds and stores observations of events associated 
with the measurement item in a combines systematic observation and assessment using pre- 
5 defined descriptor or measurement questions or items with simultaneous digital recording 

that includes a data track sequenced by a time track of time data; and (c) indexing software 
that e nt e rs an index mark based on said time data into an editable databas e d e noting said 
measurement item. Any annotation, cross r e f e r e ncing, or e ntry of notes will occur in the 
databas e in association with th e time data track ind e x without affecting the index. 

10 In us e , the testing system of the invention includ e s the following steps : (a) 

presenting a measurement item to a subject or operator; (b) digitally recording on a digital 
storage medium at least one event associated with said m e asurement item, said digital 
r ecording having a event data track of e vent data sequenced by a time track of tim e data; (c) 
indexing said digital reco r ding by entering time data related to said event into a field on 

15 compute r database along with an id e ntifi e r of said measurement item; and (d) optionally, 

augmenting said computer database by adding information to said databas e that r e lates to 
said event field without changing th e relationship of said tim e data to said event data. 

The system and method of the inv e ntion p ermi t in order to automatically generate 

meaningful indexes that support improvements in efficiency, consistency, and accuracy in 

20 retrieving and utilizing the recorded material Based upon the contents of the measurement 

questions or items, including any qualitative or quantitative descriptions or numerical rating 
results from use of the descriptors or measurement items to be associated to observed 
events, the responses reco r ded and indexed by measurement it e ms onto a computer-based 
database for review, annotation and augmentation of diverse information without affecting 

25 the time sequencing of the recorded data o r the ability to replay a portion or all of the 

re corded events. Such a system will allow assessors and others using a "test and response" 
m e thod of investigation much greater note^taking and annotation flexibility than was 
previously available with p rio r systems. The system of the invention also permits assessors 
to re-orde r, the indexes can readily be further processed and manipulated to additional 

30 advantage. The invention provides a method for access and manipulation of digital 

recordings of events and compare similar e vents from diffe r ent subjects on th e basis of the 
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I associate db v use of these pre-entered or pre-existing measurement items in ord e r to 
investigate possible r elationships and/o r tr e nds not pr e viously disc e rnibl e . 
as meaningful index information designators such as key words, labels, item numbers, or 
numerical ratings. Such a method is particularly useful for processes relating to systematic 
interview or assessment methods, including the training and monitoring of interviewers or 
assessors and the storage, retrieval, analysis, and other manipulation of recordings. 

Brief Description of the Drawings 

FIG. 1 is a flow diagram of the reco r ding an d ln the drawings, closely related figures 
have the same number but different alphabetic suffixes. 

Figs. 1 A and IB show an example of a measurement item from the World Health 
Organization Schedules for Clinical Assessment in Neuropsychiatry. 

Fig. 2 is a generalized block diagram illustrating the configuration of the systematic 
assessment and simultaneous recording and automatic indexing functions of a preferred 
embodiment of the present invention. 

7 igs. 3 A and 3B illustrate further details of the systematic assessment and 



simultaneous recording and automatic indexing functions of a preferred embodiment of the 
present invention. 

FKrig. 24 is a flow diagram of the playback and editing flmctions block diagram 
illustrating the configuration of the search, sort, retrieval, playback, and manipulation 
functions of a preferred embodiment of the present invention. 

Fig. 5 is a block diagram illustrating the configuration of post-processing systematic 
re-assessment functions of a preferred embodiment of the present invention. 

Figs. 6A and 6B show screen shots of a preferred embodiment of the present 
invention. 

Detailed Description 

The present invention is a means and method for automatically generating 
meaningful indexes for recorded material. In the following detailed description, numerous 
specific details are set forth in order to provide a thorough understanding of the present 
invention. However, it will be apparent to one of ordinary skill in the art that these specific 
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details need not be used to practice the present invention. In other instances, well known 
structures, interfaces, and processes have not been shown in detail in order not to 
unnecessarily obscure the present invention. 

A preferred embodiment of the invention will now be described herein below with 
reference to the drawings. To explain clearly, explanation will be made in accordance with 
the following order: 

(1) with respect to conducting an assessment interview; 

(2) with respect to playing back an indexed interview; 

O) with respect to searching and retrieving indexed materials; 

(4) with respect to re-assessing a prior assessment interview. 
(1) Conducting an assessment interview 

Figures 1 A fitem) and IB (rating scale) illustrate a typical measurement item from 
the World Health Organization (WHO) Schedules for Clinical Assessment in 
Neuropsychiatry (SCAN). In general, measurement items typically are composed of: 1) 
definition of a phenomenon of interest: 2) instructions regarding observation of the 
phenomenon: and 3) a quantitative rating scale. Suitable measurement items can vary over 
a wide range and will depend on the information sought from the test subject and the type of 
analysis intended. Examples of analysis include behavioral responses to audio and/or visual 
information (questions, directions, lights, sounds, etc.). physical examinations of laboratory 
specimens, oral depositions of witnesses, etc. The SCAN represents a detailed instrument 
for clinically-based, standardized psychiatric assessment of psvchopathology phenomena. 
The SCAN contains over 1 .500 measurement items, each of which is composed of at least 
an item number 102. an item title 104. and an item rating scale 120 (Tig. IB) with rating 
scale text 122 and rating scale quantitative ratings 124. and optionally an item definition 
106. an item glossary entry 108 providing further definition, suggested probes 110 
(questions) for interview use, and other instructions 112. all of which combined are referred . 
to as the "content" of a SCAN measurement item. Each measurement item as a whole with 
its constituent content may be considered as an object with respect to its function for 
conducting an assessment interview and indexing a simultaneous recording. 

Figure 2 shows a generalized block diagram of a preferred embodiment of the 
present invention. In this typical architecture, the system display presents a set of pre- 
defined measurement items 202 to the user, while simultaneously processing and digitally 
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recording the observable event to be rated, in this case the psychiatric interview 204. As the 
user carries out systematic observation and assessment with selection of a measurement item 
206. optionally generating measurement result data by using the rating scale of the 
measurement item, the system automatically defines a segment of recorded material 208. 
based upon the repeated 214 selection by the user of measurement items from the displayed 
set over the course of observation and assessment. The end of the previous segment and the 
start of the next segment is defined when a measurement item is selected. For the first 
segment no prior segment end is defined, and the start of the segment is defined by the start 
of the digital recording. Each segment is indexed to the currently selected measurement 
item that defines the start of the segment 210. The combined segment definition and 
segment indexing functions 212 are carried out in the background without user observation. 
The digital recording, measurement result data, and index information are stored in a data 
base 220. The index information thus includes the text content 102, 104, 106. 108. 110, 
112, and 120. of each associated measurement item and the rating scale 120 results 122 and 
124. which can be used as a common set of key words, labels, and quantitative attributes to 
identify index entries and allow retrieval of diverse recorded event data from different 
subjects under a common framework. The resulting assessment forms a data base file that 
can be stored, shared, or integrated into other digital records as needed by the user. 

Figures 3 A and 3B illustrate further details of the systematic assessment and 
simultaneous recording and automatic indexing functions of a preferred embodiment of the 
present invention. The system provides the user with a set of pre-defined measurement 
items and means for navigation and selection 302. Simultaneously, a process of digital 
recording occurs 304. When a measurement item is selected 306. the system automatically 
gets lower-level address information such as a time code to identify the start of the 
recording segment 308. and links the segment start address with the current measurement 
item 310. 

While these addressing and linking processes occur in the background, the user 
assesses the subject, optionally choosing to use the rating scale from the current 
measurement item to generate measurement rating data 312. When the next measurement 
item is selected by the user 314. the system automatically gets lower-level address 
information to identify the end of the recording segment 316. creates an index item by 
linking recording segment start and end identifiers to the employed measurement item. 
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along with any measurement results data 320, and then stores the index item in the data base 
220. If the measurement item was previously used, and therefor there is an existing index 
item 324 using that measurement item, then the recording segment start and end identifiers 
are appended 328 to the existing index item and stored in the data base 220. 
5 Figure 4 is a block diagram illustrating the configuration of the query, retrieval, 

manipulation, playback, and indexing functions of a preferred embodiment of the present 
invention. One means of retrieving recorded material is through use of the pre-defined 
measurement items 402 that were used in the creation of the index items. From the display, 
the user can view and select a measurement item. Likewise, the data base can be queried by 

10 searching the index items for key words from the content of the associated measurement 

items 404. The data base can also be queried by searching for specified assessment results 
406. From these actions, either singly or combined 408, the system automatically generates 
a query 410 to the data base 220 that retrieves 412 the associated segment or segments of 
recorded material for further management 416. Any search results can be sorted or filtered 

15 using standard means, and can be generalized to search across multiple subjects and 

multiple assessments using standard means. Retrieved recorded material can be selected 
and manipulated for output to another system or file, or can be played back with standard 
functions 418. During playback, additional indexing can be performed. If further indexing 
is desired by the user 420. the system supports a choice 422 of creating and using new index 

20 labels 424 or using existing measurement items 426. The indexing process 428 described 

earlier associates these index elements with recorded material, and the results are stored 430 
in the data base 220. 

Figure 5 is illustrates the configuration of post-processing systematic re-assessment 
functions of a preferred embodiment of the present invention. A re-assessment mode 502, 

25 re-assessment subject 504, and an existing assessment 506 can be selected 502. allowing the 

observation through playback of the recorded materials association with the existing 
assessment 506. The set of pre-defined measurement that was used to carry out the original 
assessment is displayed 508, allowing its use for navigation and selection of specific 
indexed segments of recorded materials. Standard playback functions are also available 

30 512. The materials observed through navigation and playback can then be re-assessed using 

the set of pre-defined measurement items 514, and the re-assessment results stored 516 in 
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the data base 220. Standard management functions can then be used to process the stored 
data 520. 

Figures 6A (assessment) and 6B (playback) illustrate a preferred embodiment of the 
present invention as implemented in a working software program ("CliniMetrica"). For 
assessment, a set of pre-defined measurement items is displayed 602. from which the user 
can select the current measurement item 604. The rating scale text 608 and quantitative 
values 606 allow systematic rating. Digital audio and video recording can be started 620, 
and will be automatically indexed in sequence with the selected measurement items. For 
playback, the list of measurement items used in the assessment is displayed 632, allowing 
retrieval of the correlated segment of recorded material, for playback via a standard set of 
playback controls 640. Further, the indexed recorded material can be retrieved through text 
search of all of the measurement item, including rating scale, text 638. The recorded 
material that is played back can be re-assessed 650. 

The invention is described in detail with reference to inquiries from an assessor and 
responses made by an interviewee during an oral intervie w assessment . It will be 
understood that the description is equally applicable to any form o f systematic assessment or 
observation by verbal, visual, audio, or physical method (e.g., questions or directions 
whether presented orally or in printe dtext form, mental or physical tests or testing, use of 
software or hardware, etc.)- and any form of response by the subject, human or other, of 
observation (verbal, nonverbal, or physical response) with mere changes in semantics to 
describe the particular form of test and response. 

A general testing system according to the invention includes the following steps: (a) 
presenting a set of pre-defined measurement items to a subject or operatorrf H for use in 
structuring and carrying out the observation and assessment of events or phenomena of 
interest; (b) simultaneously making a digital recor drecording of one or more aspects of the 
observed events or phenomena on a digital storage medium at l e ast one event associated^ 
with said measurement item, said digital r e cord havin g recording including an event track of 
event data sequenced by a time track of time data; (c) automatically indexing said digital 
recording by automatically entering time dat a and an identifier relating to the current 
selected m easurement item into a fields o nr ecord in a computer database data base ; and (d) 
optionally, augmenting said computer database data base by adding information to said 
database data base that relates to said event field measurement items or said digital recording 
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without changing th epreviouslv established indexing relationship of said time s and data to 
said base records, ev e nt data. 

An interviewin g assessment system according to the invention includes a pe r sonal 
computer having: (a)- an output communication subsystem that allows communication with 
5 an interviewee! or interviewere so tha t pre-defined measurement items can be presented; 

and selected (b) an input subsystem for the interviewer or interviewee to respond to the 
measurement item communications to generate assessment data results : (c) a recording 
subsystem that simultaneously m akes a digital record of the output communications as well 
as th e interviewee ? s responscs audio. video, physiologic, or other types of recordable event 

10 data from either or both of the interviewer's and interviewee's behaviors and responses. 

with a data track o f said event data synchronized byto a time track of time data; (d) a-dtgital 
storag en indexing subsystem that aHow sautomatically generates correlation entries that 
define time segments of the digital r ecord to be stored for playback and editing; and (e) a 
computer with database softwar e that can generate a database event index record based on 

15 said time data for cac hr ecording based upon the selection or presentation of successive 

measurement ite mitems. and links those time segments to the measurement items, thereby 
creating a meaningful index for these segments of digital recording by use of the contents of 
the correlated measurement items and any quantitative measurement rating results: and (e) a 
digital storage retrieval subsystem that uses the measurement item-based index to retrieve 

20 segments of recorded material for playback, analysis, editing, annotating, re-rating, or other 

related purposes . 

The means for presenting the subject or operator with measurement items can take 
many forms^ depending on the type of measurement item involved with the assessment. The 
system and its method of use is most easily described here in the context of an interview 

25 between a subject and an interviewer using a computer to display a series of measurement 

item questions to the interviewer as a guide for the interview. ^At the same time, t he 
interview is recorded by a audio or video recorder (preferably digital) that includes a data 
track synchronized by a time track of time data. As the interviewer moves from one 
question (i.e.. measurement item) t o anothe ra new question via input tointo the computer, 

30 th ethis question-switching input is an act that causes the databasc data base (either from 

software or hardware functions') to record the time data (preferably a stop data bit for 
termination of the preceding time segmen t of recording and a start data bit for beginning the 
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I next time segment of recording) and an associated identifier that pcrtains links the time 
| segment of recording to the new question (measurement item V thereby generating an index 
| entry . 

As those skilled in the art will appreciate, the indexing act can be done in an active 
interview situation, by the interviewer, as questions are posed. The indexing can also be 
done in passive situation, where the operator is observing but not necessarily actively 
| interacting, by the operator entering mrinputg into the computer from a list of predefined 

topic items (measurement items) during actions by the subject of observation. Either 
| method will generate a database data base of event data that is indexed by a series of 
| measurement item-based identifiers that can be sorted and retrieved for selective retrieval 

and playback of the events. 
| Preferably. thc The preferred software embodiment of the invention for a personal 

| computer tha twill hasve the capability to: (i) communicate pre-defined inquiries to the 
| interviewee directly or through an interviewer in a systematic manner;; (ii) provide ttrating 
| scaleg or method methods associated with or part of the pre-defined inquiries by which said 
| inquiries can be rated , and Cm) provide or applied for the purpose of describing said 
| interviewee: and Cm) automatically generate an index in the form of retrievable; index marks 
| associated with the pre-defined inquiries. A personal computer is used to communicate the 
| inquiries and their toat the same time capture and digitally record or control a recording 
device that records the interviewee's responses to form a digital record of indexed inquiries 
and indexed responses. 

| During later p layback and i f As those skilled in the art will appreciate, any type of 

| computing device could be used to provide these functions, including customized or 
| programmable hardware that provides functions described above as provided by software. 
| If digitally recorded responses are to be further indexed during later playback , the 

| system can also insert additional response i ndex marks and the identifiers for the mthe index 
| marks into the digital record so that r esponses arc digital ly marked materials are further 
| indexed for retrieval. An assessor may also insert additional inquiry index marks into the 
| recorded interview. Index F or example, in reviewing an earlier interview, an assessor may 
| wish to employ pre-defined measurement items not used originally in interview, to further 
| assess or rate the interview. In addition to use of pre-defined measurement items which 
| provide meaningful text content that serves as index labels or key words, new index marks 
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can be placed at any point in the recording but an inquiry index mark is pr e ferably 
inserte d with manual entry of a label or key words. Furthermore, if the user wishes for 
example to study or analyze communication and response patterns, new measurement items 
could be created and stored for repeated use. Preferably, an "inquiry" measurement item 
5 could be created, stored, and then inserted at each location of interest where a specified data 

item inquiry is being communicated, and a r es p onse index ma r k is "response" measurement 
item could be created, stored, and inserted where the response is made. The inquiry and 
response index marks are desirably standardized and relate to the systematic content of the 
inquiry or the response. 

10 The invention is conveniently explained in the context of an interview under the 

WHO Schedules for Clinical Assessment in Neuropsychiatry (SCAN). These schedules 
represent a detailed instrument for clinically-based, standardized psychiatric assessment of 
psvchopatholog y phenomena . The SCAN contains over 1.500 measurement items, each of 
which is composed of at least an item number, an item title, and an item rating scale, and 

15 optionally an item definition, an item glossary entry providing further definition, suggested 

probes (questions) for interview use, and other instructions, all of which combined are 
referred to as the "content" of a SCAN measurement item. An example measurement item 
from the SCAN is shown in Figs. 1 A and IB. The SCAN is a widely recognized and used 
international standard , having been translated into over 25 languages . It will be understood 

20 by those skilled in the art that the techniques applied to a SCAN assessment would be 

applicable to other types of assessment instruments, physical examinations, or other 
physical phenomenon whose analysis is aided with a recording of sights, sounds* 
physiology , and/or other domains of observation with only minor semantic differences that 
describe the same essential types of events. 

25 During an Using the present invention to conduct a SCAN psychiatric assessment 

interview with a human subject, real-time selection ofb^ the interviewer of a relevan t SCAN 
item in the natural course of the evetrt sinterview. observation, and assessment of clinical 
phenomena automatically generates a s e ries of time-stam p ed inquir y time-based index 
mark sentry link that scrv callows use of the SCAN item as an informative indicum for the 

30 simultaneously r ecorded dat a (e.g.. any of audio, video, physiologic) . The time-stamp index 

marks (time stamp plus identifier) are stored in databasc data base fields with time data that 
is correlated to a sequence of event data in the recording of the events. The resultant set of 
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time stamp index marks then provides a general index structure which can be used to further 
support, independently of the real-time process, a flexible note-taking structure and user 
interface that complements diverse personal note-taking styles and application needs. 
Notations and annotations in the form of text, sound, video, or other data can be 
5 | entered into the database data base as separate fields of information without affecting the 

relationship between the time stamp index mark and the recorded data associated therewith. 
| Thus, the use of digital index marks of an event recording in an editable database data base 
provides assessors with the ability to generate assessment records rich in diverse content. 
When coupled with standardized assessment inquiries (e.g., a list of standard interview 
10 | questions) and a databas e data base field that identifies each inquiry by a standardized 
descriptor for each inquiry, this rich-assessment record can be used as a basis for 
comprehensive comparisons on a rich, diverse content. 

In the case of assessment interviews in which there is no pre-existing set of 
| standardized measurement items (questions, definitions, inquiries, instructions, directions, 
15 or other tasks), an important feature of the present invention is the ability for the assessor or 

| another individual (e.g., an interview designer) to generate a set of items prior to the event 
| to be re cord e d and d e v e lop a new set o f observed, assessed, and recorded. This supports the 
| process of developing a new set of standardized inquiries for investigating new issues. This 
set of items can be used and re-used as real-time index information designators which can 
20 later be used as standard access mechanisms into both the notes and the correlated recorded 

signals. These new inquiries, e.g., key words, labels, or any other pointer to information, 
| can identify specific concepts in the user-produced information fbrto which the user wants 
| to have later access. An information designator is treated as an object that t scan be easily 
| created and reused-ancK. This novel implementation both in the user interface and data 
25 | structure orf the present invention provides a flexible and powerful indexing tool tittofcr the 
recorded signals in addition to the access provided by the recorded medium's address 
| parameters. 

| In an environment where a recording of events is being made simultaneousl y with a 

| the rating of those events, the interviewer has limitations , due to limitations of human 
30 | cognitive abilities, on what actions can be taken or what additional items can be considered 
| as the interview is in progress. The present invention produces a digital database data base 
| of indexed inquiries and responses that can be used to organize the assessor's notes. 
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originally written in free text or graphical fornix but later entered into the databasc data base 
with minimal restriction as to the amount of that information or when it is entered. The 
inquiry and/or response index marks allow the assessor's notes to be correlated with the 
recorded signals of the event. = Further, the present invention provides the assessor with the 
5 ability to modify any notes that were previously entered without regard to when the 

modifications are made or where they are located on the recording medium. 

The invention recognizes, moreover, that the temporal or spatial sequencing of 
information is itself often a significant clue as to the content of the information and may aid 
a user in recalling memories or in retrieving recorded information from a perceived event. 

10 The inquiry and response index marks of the present invention provide a mechanism for the 

user to control the playback sequence and thereby view the index with items sorted in, e.g., 
chronological order, type of inquiry (as denoted by the type of inquiry index encoded into 
the digital record), by type of response (also as denoted by the type of response index mark 
encoded into the digital record), as well as any other order desired by the assessor. 

15 The system of the present invention is particularly effective for video and audio 

interviews where the events include a time sequence of responses by the interviewee or test 
subject. These events are recorded as a series of audio or video input dat a segments 
sequenced by a time track of time data. Because the amount of detail that may be captured 
for a particular time on an associated recorded medium is open-ended, a richer set of 

20 annotations may be efficiently made by use of pre-defined measurement items while the 

video is being digitally recorded in reaNtime than is typically allowed in previous analog 
systems. This relieves the note-taker (i.e. the measurement assessor) of many of the 
pressures typically associated with taking interview notes in real time. The time 
zon csequence and reusable information designator object features provide for treating the 

25 event as a whole bv permittin g and permit later augmentation or modification of previously 

made notes. This relieves the note-tak e r of many of the p r essures typically associated with 
taking interview notes in real time. 

It is another significant advantage that the system of the present invention operates 
in both real-time and post-processing environments. This feature recognizes that note- 

30 taking as a human activity is typically an interactive process. The output data structure 

(e.g., the indexed video kr gmaterials) : produced by the system of the present invention 
during a real-time note-taking session while the event is being recorded may be further 



25 



modified and/or augmented by the same user or by different users during post-processing 
review sessions occurring in parallel or scriatim serial . This facilitates and potentially 
reduces the time involvcd required hrthe post-processing sessions. 

A system for correlating event data being recorded in real time to event information 
entered into the system by a system user includes input circuitry connected to a user input 
device for producing signals indicating actions of a system user; output circuitry connected 
to a display having area for presenting images; and a processor connected for receiving the 
signals from the input circuitry, and connected for providing images to the output circuitry. 
The system also includes a digital recording device in communication with a digital storage 
medium for producing a plurality of signals indicating recorded event data about an event 
for storage on the storage medium. The recording device is currently producing the 
plurality of signals and indicating the recorded event data and is connected for 
simultaneously providing address data items to the processor of the syste m and to the 
recording device . The recording device stores the plurality of signals indicating the 
recorded event data on the storage medium so that a recording address value of each address 
data item provided by the address source indicates a portion of the recorded event data. The 
systems also includes memory for storing data 5 including instruction data indicatin g that 
indicate instructions the processor executes. 

Stated another way, the system may be configured to operate in a post-processing 
environment so that an event that has already been recorded but not indexed is the event 
about which a log is constructed according to the present invention. The event that has 
already been recorded is digitally stored on a storage medium as a plurality of signals 
indicating recorded event data about the event. In this aspect of the invention, the system 
includes an output device for presenting the previously recorded signals thereon to the 
system user. This output device is in communication with the storage medium including the 
plurality of signals indicating recorded event data about the event. The output device is 
connected for providing a plurality of address data items to the processor of the system as 
user-requested address values; each address data item is a recording address value indicating 
a portion of the recorded event data. Preferably, the plurality of signals indicating the 
recorded event data includes the plurality of address data items, but the output device may 
itself generate them as the plurality of signals is being presented by the output device. 
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The system of the present invention permits a system user to interactively produce 
stored information that is correlated with an "address". The "address" refers to a 
mcasurcmcn ts equence dimension used to identify a portion of the stored information. When 
the stored information is being correlated with recorded signals, the address typically refers 
5 to the location of the recorded signals on the storage medium storing the recorded signals. 

However, the system of the present information is not limited to correlating information 
with recorded signals, and may be used in any situation where later access to the stored 
information would be facilitated by correlation with an address, and by having an access 
mechanism as provided by the "information designators". Generally, when the system user 

10 is producing a database data base of stored information, the address record will be time data 

taken from the time track of the recording of the events. 

The computer-controlled system of the present invention uses common, readily 
available hardware components. For an assessment interview delivered via computer, the 
system includes a computer with an image display that will display questions or directions, 

15 an input that receives signals from a user interaction device, e.g., keyboard, digital tablet, 

microphone, camera^ or other recording device that produces a data track synchronized by a 
time track. The display can present the measurement item questions directly to the subject 
or, alternatively, can display a series of questions to an interviewer/assessor who then 
presents the questions to the subject. Some act taken to move from one question to another 

20 generates the database data base time entry or start/stop time entries as well as the index 

identifier that pertains to the question. 

Analog recording devices can be have their recordings digitized^ o r the observable 
events can be recorded directly as a digital recording that includes: (a) a data track of sound, 
image, or other event data that are synchronized by (b) a time track that includes time data 

25 correlated with event data. The use and existence of these two data tracks is typically 

transparent to the user of the equipment but can be read by a computer and used to generate 
databas e data base entries for each recorded event based on the time track data associated 
therewith from the digital recording. 

The user interaction devices useful in the present invention may include any one of a 

30 variety of input devices controllable by a human user. For example, the user interaction 

device may include a pointing and gesturing device to indicate actions, gestures, and strokes 
such as those produced when making handwriting gestures. A stylus or other form of 
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handheld pen are examples of a suitable user interaction device. Other pointing devices 
| such as a mouse o r a t r ackball , trackball, eve position, or brain activity electrical or 
| magnetic patterns may also be suitable for indicating actions, gestures, and strokes. 

The present invention is furth e r unde r stood by ref e r e nce to FIG. 1 which shows a 
5 flow diag r am of the recording and indexing functions of the pres e nt invention. A digital 

r ecording 1 of a scries of events is mad e directly by a digital record e r or made into a digital 
formation 2 from an analog reco r ding. The indexing system 3 is used to generate a database 
4 of index items 5 based on measurement items 6. 

Suitable measur e ment items can vary over a wide range and will depend on the 

10 information sought from the test subj e ct and the ty p e of analysis intended. Examples of 

analysis include behavioral responses to audio and/or visual information (questions, 
dir e ctions, lights, sounds, etc.), physical e xaminations of laboratory specim e ns, oral 
depositions of witnesses, etc. In the cas e of a behavioral examination, measurement items 6 
can be generated in process 10 from a standa r d assessment instrument 7, open ended 

15 questions 8 and any oth e r inputs 9 desired by the assessor. Preferably, process 10 is a 

simple system that is readily repeated. Suitable examples include a question outline or list. 

Similarly, FIG. 2 shows a flow diag r am of the playback and e diting functions of the 

present invention. Digital data 20 mad e of an event sequence track of e v e nt data is 
synchronized by an associated time track of time data is played back 21 for audio, video, 

20 and/or physiological events. Preferably, n e w index items 22 are c r eated during a post" 

processing playback and review phas e but may also be created separately, for example as a 
pre-existing detail e d set of measurement sub-items, o r as additional notes during the o r iginal 
assessment process. Each index items is bas e d on a link 23 between the time data in the 
time track with corres p onding e vent sequ e nc e data and entered as index data 24 in a 

25 database. This database can include associated r ecords that identify the index entries by a 
common set of labels, ind e x attributes 25 o r othe r identifier tha t allow diverse event data 
from different subj e cts under a common fram e work. The resulting assessment forms a 
database file 26 that can be stored, shared, or integrated into other digital reco r ds as needed 
by the assessor. Although the foregoing disclosure relates to preferred embodiments 

30 of the invention, it is understood that these details have been given for the purposes of 

clarification only. Various changes and modifications of the invention will be apparent, to 
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one having ordinary skill in the art, without departing from the spirit and scope of the 
invention. 
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